SEPIA: Surface Span Extension to Syntactic Dependency Precision-based MT Evaluation

We present a new Machine Translation (MT) evaluation metric, SEPIA. SEPIA falls within the class of syntactically-aware evaluation metrics, which have been getting a lot of attention recently (Liu and Gildea, 2005; Owczarzak et al., 2007; Giménez and Màrquez, 2007). Specifically, SEPIA uses dependency representation but extends it to include surface span as a factor in the evaluation score. The dependency surface span is the surface distance between two words that are in a direct relationship in a dependency tree. The basic idea behind SEPIA is that long-distance dependencies should receive a greater weight in MT evaluation metrics than shortdistance dependencies. This is because we suspect that having more long-distance matches indicates a higher degree of grammaticality. In the rest of this document we describe the SEPIA metric and its variants, and the publicly available SEPIA package.